Lecture 2 The Chernoff bound and median-of-means amplification

In the previous lecture we introduced a streaming algorithm for estimating the frequency moment F2. The algorithm used very little space, logarithmic in the length of the stream, provided we could store a random function h : {1, . . . , n} → {±1} “for free” in memory. In general this would require n bits of memory, one for each value of the function. However, observe that our analysis used the fact that h is a random function in a rather weak way: specifically, when computing the expectation and variance of the random variable Z = c describing the outcome of the algorithm we used conditions such as E[YiYjYkY`] = E[Yi] E[Yj] E[Yk] E[Y`] for distinct values i, j, k, `, where Yj = h(j) is the random variable that describes the output of the function at a particular point. As it turns out, this requirement is a much weaker requirement than full independence, called 4-wise independence. In particular, it is possible to sample “4-wise independent” functions using much fewer random bits than a uniformly random function. This is the idea behind derandomization: to try to save on the number of random coins needed while keeping the function h “random enough” that the analysis carries over.